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Assessing Civic Competency and Engagement in Higher 
Education: Research Background, Frameworks, 
and Directions for Next-Generation Assessment 


Judith Torney-Purta , 1 Julio C. Cabrera , 2 Katrina Crotts Roohr , 3 Ou Lydia Liu , 3 & Joseph A. Rios 3 

1 University of Maryland, College Park 

2 University of Minnesota, Twin Cities 

3 Educational Testing Service, Princeton, NJ 


Civic learning is increasingly recognized as important by the higher education and workforce communities. The development of high- 
quality assessments that can be used to evaluate students’ civic learning during the college years has become a priority. This paper 
presents a comprehensive review of existing frameworks, definitions, and assessments of civic-related constructs from approximately 
30 projects relevant to higher education, and includes a discussion of the challenges related to assessment design and implementation. 
Synthesizing information from the review, we propose an assessment framework to guide the design of a next-generation assessment of 
individuals’ civic learning that takes advantage of recent advances in assessment methods. The definition identifies 2 key domains within 
civic learning: civic competency and civic engagement. Civic competency encompasses 3 areas (civic knowledge; analytic skills; and par¬ 
ticipatory and involvement skills), and civic engagement also captures 3 areas (motivations, attitudes, and efficacy; democratic norms 
and values; and participation and activities). We discuss item formats and task types that would ensure fair and reliable scoring for the 
assessment. The review of definitions of civic learning and its components developed by organizations, the proposed assessment frame¬ 
work, and assessment considerations presented here have potential benefits for a range of higher education institutions. This includes 
institutions that currently have students engaged in relevant curricular or cocurricular activities and also institutions that would find 
assessments of civic competency and engagement helpful in program development or in evaluating students’ accomplishments. 

Keywords Student learning outcomes; higher education; civic learning; civic competency; civic engagement; assessment 

doi: 10.1002/ets2.12081 


Introduction and Rationale 

Taken as a whole, education seeks to do two things: help young persons fulfill the unique, particular functions in life 
which it is in them to fulfill, and fit them so far as it can for those common spheres which, as citizens and heirs of a 
joint culture, they will share with others. (Conant, 1945, p. 4) 

Over the past several decades, educators have made it a priority to promote a civically literate society that helps to foster 
democracy and a growing economy. It has also been a priority for many who are striving to create a more just and fair 
world. In this paper, we provide a detailed description of civic learning for students in higher education. We then break 
down this larger construct of civic learning into two key domains: (a) civic competency (i.e., civic knowledge and skills), 
and (b) civic engagement. First we introduce the topic of civic learning and suggest several reasons why it is important. 
Then we provide a detailed review of current conceptual frameworks, research, and assessments of civic learning. After 
reviewing existing frameworks and measures, the main purpose of this paper is to construct an assessment framework for 
these two key domains of civic learning that could be elaborated to guide the development of next-generation assessments 
featuring a variety of item formats, innovative task types, and online delivery with accessibility considerations for all 
students. Challenges and limitations in assessing civic competency and engagement are also discussed. 

Corresponding authors: K. C. Roohr, E-mail: kroohr@ets.org; and J. Torney-Purta, E-mail: jtpurta@umd.edu 
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The Importance of Civic Competency and Engagement in Higher Education 

Educational leaders stress the need to include learning that is related to the development of individuals’ civic capacity 
throughout all years of schooling in the United States (Conant, 1945; Dewey, 1916; Ehrlich, 1997; Pollack, 2013). They 
have examined a variety of sources of content and pedagogy in the United States, as well as in programs developed abroad. 
Recently, in a report commissioned by the U.S. Department of Education, the National Task Force on Civic Learning 
and Democratic Engagement, an initiative of the Association of American Colleges and Universities (AAC&U), made 
an urgent call to higher education institutions in the United States to make civic literacy, inquiry, and action part of the 
educational objectives to be achieved by every college graduate. This plan would involve adopting long-term measurable 
standards to indicate the extent to which college students are gaining a civic perspective during their postsecondary edu¬ 
cation (National Task Force on Civic Learning and Democratic Engagement [National Task Force], 2012). By referring 
to a “crucible moment” in the title and throughout the text, the report emphasized a convergence of issues and concerns 
over the last decade. Higher education institutions themselves have acknowledged the importance of postsecondary edu¬ 
cation in developing civic learning, with 68% of the chief academic officers surveyed from the 433 member institutions 
of the AAC&U recognizing civic engagement as an essential learning outcome (AAC&U, 2011, p. 20). A further exhor¬ 
tation appears in the National Task Force report that higher education institutions should be supported to “develop a 
national framework of civic indicators across knowledge, skills, values, and collective action” (National Task Force, 2012, 
p. 38). Recently, taking concrete steps in this direction, the Research Institute for Studies in Education (RISE) at Iowa State 
University completed a paper reviewing the literature in the area of civic learning and engagement for AAC&U and the 
American Association of State Colleges and Universities (AASCU; Reason & Hemer, 2015). 

The groups in the higher education community referred to in the previous paragraph have extended calls to action in 
reports that focused on K-12 education, such as Guardian of Democracy: The Civic Mission of Schools (Gould, 2011). This 
widely cited report included calls for postsecondary institutions to “require all students, regardless of major, to take at 
least one engaging civic learning course” and encouraged postsecondary students to “volunteer as civic mentors in K-12 
schools” (Gould, 2011, p. 43). 

Disciplinary Interest in Civic Engagement in Higher Education 

Some who work in this area prefer the adjective political to the adjective civic (or vice versa) in describing engagement. 
Taking an empirical approach, Bennion and Dill (2013) examined the terminology found in titles and abstracts in the two 
major journals publishing research on undergraduate political science instruction (i.e., Political Science and Politics and the 
Journal of Political Science Education). They found that the concepts of civic skills or engagement and service learning were 
mentioned about equally. Engagement and skills with an explicitly political focus were mentioned slightly less frequently 
than either civic engagement or service learning (Bennion & Dill, 2013). Another attempt to distinguish civic and political 
concepts comes from a latent class analysis of the types of engagement among 1,800 recent college graduates who reported 
their organizational engagements on an ACT alumni survey (Weerts, Cabrera, & Perez Mejfas, 2014). Forty percent of their 
sample (the largest cluster group as revealed by a latent class analysis) was active in civic/charity activities but avoided 
political, partisan, or social change organizations. In general, there appears to be a tendency to avoid framing definitions 
in terms of explicitly political activism (especially partisan activities) in most of the studies reviewed and a preference 
toward the term civic engagement. 

This issue also should also be considered in a more substantive way. Regardless of whether one promotes civic or 
political actions, this raises normative issues. These issues are contested among groups that advocate different civic- 
related programs. There is considerable common ground but also significant principled disagreement. The question can 
be framed in this way: On what values should programs be based? To name just a few, these values might include respect 
for the exceptional character of Americas democracy and its economic system, a participatory democracy’s need for 
high levels of conventional political participation (often assumed to be connected with partisanship), ideals of social 
justice or human rights (often fostered through programs of volunteering), or the need to encourage ethical and socially 
responsible personal behavior (Levine & Higgins-D’Alessandro, 2010; Reason, 2011; Westheimer & Kahne, 2004). The 
focus of this paper, however, is not to evaluate frameworks primarily in relation to their underlying value dimensions. It is 
more concretely to review existing frameworks, assessments, and research, and to propose a comprehensive, yet feasible 
approach to further elaborate this domain through the development of an assessment framework. The next sections 
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describe approaches to civic competency and engagement as they have been elaborated by scholars within academic 
institutions as well as employers. 

Several fields of study have mentioned civic engagement prominently in their recommendations for undergraduate 
education, including political science. Data from national samples of adults of voting age have been the source of inferences 
about political engagement going back to election studies in the 1960s (see the review in American National Election 
Studies [2015] and the landmark 1995 book by Verba, Schlozman, and Brady, Voice and Equality: Civic Voluntarism in 
American Politics). In these conceptualizations, the process of involvement requires resources (e.g., discretionary time, 
money, civic skills, and political information) along with psychological engagement in political processes and recruitment 
to become involved in political activity. Higher education plays a vital role in the development of these resources. Ten years 
after Verba et al.’s (1995) landmark book, civic engagement was the central concept in a study of generational differences 
between adolescents and adults (Zukin, Keeter, Andolina, Jenkins, & Delli Carpini, 2006). The Center for Research on 
Civic Learning and Engagement (CIRCLE), established at about the same time and now located at Tufts University, has 
focused attention on political action (especially voting) but also on civic engagement. CIRCLE considers a range of ages 
and does not have a particular disciplinary focus. 

Beginning about a decade ago, the American Sociological Association (ASA) began to elaborate the idea of public 
sociology after the concept was highlighted by Michael Burawoy in his address as the president of American Sociolog¬ 
ical Association, a presentation that has stimulated extensive commentary in the field (Jeffries, 2009). Public sociology 
attempts to make research more relevant to members of the public whose decisions could be informed by understanding 
concepts such as social power, marginalization, or social networks and by deliberating on their implications in a concrete 
situation. Cans (2009) has argued that addressing the public is an appropriate role for the sociologist, who often serves 
as “an investigative reporter and analyst of social injustice” and looks at “what is taken for granted and unexamined in 
everyday life” (p. 125). 

Engaged sociology is the term used to describe programs of civic engagement and community activity among under¬ 
graduates who are learning to apply sociological concepts and use sociological tools (Korgen & White, 2010). These 
programs sometimes rely not only on volunteering or service-learning activities but also include involvement with social 
movement or activist organizations, with other civil society groups, and with journalists or media specialists. 

In summary, the disciplines of political science and sociology, through general education courses as well as the prepa¬ 
ration of majors, are in the forefront of enhancing young people’s overall civic capacity, but they are not alone. History 
departments are increasingly offering (and sometimes requiring) courses on the history of democratic institutions, social 
movements, and civic action (e.g., James Madison University, 2015). Humanities departments, including departments of 
English, have recently shown interest in civic engagement (Grobman & Rosenberg, 2015). Tosh (2014) asserted that citi¬ 
zens’ abilities to examine issues of public interest in their historical contexts are essential in a thriving democracy (and he 
invoked the concept of public history). Additionally, the American Psychological Association (APA) has taken a positive 
stance toward activities that foster students’ action and sense of responsibility in the community (APA, 2013). Finally, 
there has been considerable attention to the “civic-minded graduate” who develops competence and engagement regard¬ 
less of his or her major field (Steinberg, Hatcher, & Bringle, 2011; Steinberg & Norris, 2010). Required general studies 
courses and cocurricular activities are expected to contribute to the civic-minded graduate’s political capacities. 

Employers' Interest in College Graduates' Civic Competency and Engagement 

The value of colleges and universities promoting the development of civic-minded individuals has also been recognized as 
contributing to the quality of the workforce. Employers often report that the technical skills that have dominated the 20th 
century are important (especially for those entering science, technology, engineering, and mathematics [STEM] fields), 
but these skills are not sufficient for prospering in the global economy of today. Employers in the 21st century are seeking 
to hire and promote individuals with knowledge of significant changes in society, intercultural literacy, ethical judgment, 
humanitarian values, social responsibility, and civic engagement (Casner-Lotto & Barrington, 2006; Gould, 2011; Hart 
Research Associates, 2010, 2013, 2015; Peter D. Hart Research Associates, 2006, 2008). In fact, according to a recent 
survey conducted by Hart Research Associates (2015) on behalf of the AAC&U, 87% of over 400 employer respondents 
stated that all students, regardless of major, “should gain an understanding of the democratic institutions and values” 
(p. 4). Additionally, 86% of respondents stated that students should “take courses that build the civic knowledge, skills, 
and judgment essential for contributing to a democratic society” (p. 4). 
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A second way in which civic competency and engagement have been related to workplace readiness is through studies of 
organizational citizenship behavior (OCB). Organizational psychologists define OCB as individual employee’s “behavior 
that is discretionary, not directly or explicitly recognized by the formal reward system, and that in the aggregate promotes 
the effective functioning of the (employee’s) organization” (Organ, 1988, p. 4). The civic virtue dimension of OCB pertains 
to employees taking an active interest in improving the social and psychological environments of the organizations in 
which they work. A meta-analysis of studies with more than 50,000 respondents showed significant associations between 
OCB scale scores and lower likelihood of employee turnover, as well as higher productivity at the organizational level 
(Podsakoff, Whiting, Podsakoff, & Blume, 2009). 

Given the value employers place on civic-minded individuals entering the workforce (in addition to the disciplinary 
groups that support these aims), a civic-related strand of postsecondary education appears to have considerable potential. 
In fact, attention to civic competency and engagement is particularly appropriate in higher education because this is a 
developmental period when students are choosing career paths and acquiring both specialized knowledge or skills and 
the behaviors required to succeed in a job and as a citizen or member of the community. A review of civic missions across 
higher education institutions concluded that civic development is both a public good (i.e., enhancing the community 
and political or civic institutions) and a private good (i.e., enhancing employability and providing intrinsic satisfaction to 
individuals; Carnegie Foundation for the Advancement of Teaching & CIRCTE, 2006). 

The Need for a Coherent Set of Definitions 

Even though there is agreement about the importance of civic learning, little shared language exists for labeling its dimen¬ 
sions in a way that could serve as the basis for developing a next-generation assessment. A number of labels (e.g., civic 
learning, civic capacity, civic education, citizenship) and competencies (e.g., civic skills, civic inclinations) have been 
proposed by professional organizations, governmental agencies, researchers, and institutions of higher education when 
referring to civic learning (e.g., Markle, Brenneman, Jackson, Burrus, & Robbins, 2013). The lack of a coherent definition 
has also been recognized as a general problem. Finley (2011) concluded, “It cannot be expected that students (or faculty) 
are responding to the same set of conceptual ideas [about civic engagement] when taking a survey, writing a journal or 
responding to an interview” (p. 18). In one of the influential volumes in the Bringing Theory to Practice monograph series, 
Finley has further argued that “most of what we know about the empirical effects of civic engagement comes through the 
lens of service learning” (Finley, 2012, p. xvi). That limits the generality of findings, although she also noted that “regardless 
of whether civic engagement is defined as service learning or democratic skill building, there seems to be broad agreement 
on best practices (e.g., reflection, high levels of interaction ... and real-world applications)” (Finley, 2012, p. xvi). 

Additionally, a number of challenges are associated with measuring an individual’s civic competency and engagement. 
A considerable number of the existing assessments of civic competency and engagement in higher education have psycho¬ 
metric weaknesses, with many being self-report surveys that lack strong validity evidence. In a meta-analysis. Bowman 
(2011) found that self-reported gains in civic- and diversity-related attitudes were substantially larger than the gains mea¬ 
sured when assessments were conducted over time. This study, along with a broader review, led Reason and Hemer (2015) 
to conclude, “Civic learning research has predominantly been based on student self-report and cross-sectional design. The 
addition of more direct measures of civic learning, especially those that can be applied longitudinally, would strengthen 
the current understanding of how college experiences affect civic learning” (p. 33). 

The number of quality assessments in this area has been increasing in higher education, as individual institutions as 
well as centers and projects have developed measures (see Beaumont, Colby, Ehrlich, & Torney-Purta, 2006; Hurtado & 
DeAngelo, 2012; Hurtado, Ruiz, & Whang, 2012a, 2012b; Office for Standards in Education, 2003). The issue of psycho¬ 
metric quality will be discussed later in sections on reliability and validity. Concerns about socially desirable answering 
patterns to self-report questions, which may make respondents appear more civically engaged than they actually are, will 
be considered. 

As illustrated by the previous discussion, it is an appropriate time to look at the variety of ways in which civic com¬ 
petency and engagement have been defined and assessed across the wide range of higher education institutions in the 
United States. There are growing calls for recognition of students’ achievements in this area. This includes suggestions to 
award campus-based certificates or to offer structured course programs leading to a college minor (Butin & Seider, 2012) 
and/or digital badges, an effort explored by CIRCLE supported by the Bechtel Foundation (Sullivan, 2013). In particular, 
Holland (2014) has persuasively argued that at this time of rapid change in higher education—in its economic models, the 
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diversity of its students, the modes of teaching, and the criteria associated with institutional reputation—the field needs 
to move toward coherent and shared definitions of terms such as civic engagement, civic motivation, and civic achieve¬ 
ment. Furthermore, it is an appropriate time to exert leadership in designing a process of institutional or program-level 
assessments that colleges and universities could use to examine their own campuses and/or to recognize students’ civic 
competency and engagement. 

In the subsequent sections of this paper, we provide a review of the current frameworks, research, and assessments in 
the area of students’ civic learning, and propose an assessment framework with considerations for the design of a next- 
generation assessment. The term civic learning is sometimes used as a higher-level descriptor to integrate knowledge, 
intellectual, and participatory skills, values, and dispositions or attitudes (Gould, 2011; Hurtado et al., 2012a, 2012b; Musil, 
2009; U.S. Department of Education, 2012). In the remainder of the paper, we acknowledge the overarching construct 
of civic learning while distinguishing between civic competency (i.e., knowledge and skills) and civic engagement (i.e., 
motivation, values, and participation). 

Current Frameworks, Definitions, and Assessments of Civic Competency and Engagement 

Professional organizations, governmental entities, think tanks, scholars from universities, and experts from foundations 
have provided definitions and frameworks in an attempt to establish more coherent approaches to constructs related to 
civic competency and engagement at all levels of education. Internationally, especially in Europe, definitions and frame¬ 
works have also been developed, and assessment initiatives have been led by large-scale testing organizations such as the 
International Association for the Evaluation of Educational Achievement (IE A; headquartered in Amsterdam) and by the 
Qualifications and Curriculum Authority (QCA in the United Kingdom). 

Table 1 presents more than a dozen definitional frameworks, primarily from organizations with an interest in higher 
education in the United States. These frameworks of civic-related constructs will be discussed, highlighting both their 
similarities and differences. Table 2 presents a structured summary of assessments measuring constructs in the categories 
of civic competency and civic engagement. The majority of the organizations whose conceptual frameworks are found in 
Table 1 also appear together with some specifics of their assessments in Table 2. In other words, the entries in Table 1 were 
in most cases intended by their authors for use both as frameworks to develop programs and as guidelines for assessments. 
However, a number of frameworks have also been developed for the purpose of guiding instrument or assessment design 
and not primarily for program guidance. Frameworks that fall into this category are found only in Table 2 (e.g., National 
Assessment of Educational Progress [NAEP] Civics Assessment, National Survey of Student Engagement [NSSE] Topi¬ 
cal Module on Civic Engagement). Both tables provide relevant information to guide the development of a conceptual 
definition and a next-generation assessment, as well as ideas about modes and topics for assessment. 

Foundational Frameworks of Civic Competency and Engagement in the United States 

Beginning in the mid-1990s, scholars such as Ehrlich (1997) highlighted the lack of research on the relation of higher 
education and civic engagement and described some avenues, components, and strategies that institutions of higher edu¬ 
cation could use to remedy this situation. Ehrlich’s vision was exemplified in the Political Engagement Project (PEP) at 
the Carnegie Foundation for the Advancement of Teaching from about 2000 to 2007. Saltmarsh (2005), a scholar who 
studies the ways that engagement for democracy could transform higher education, defined civic learning as the learning 
and development of an ability for effective civic engagement by the process of acquiring knowledge (e.g., historical and 
contemporary), skills (e.g., civic imagination and creativity), and values (e.g., justice) through college courses that focus 
on democratic societies, as well as other experiences on campus and in the community. 

A number of other scholars and organizations have also put forth conceptual frameworks and learning outcomes of 
civic learning, such as the recent work of the AAC&U culminating in the publication titled A Crucible Moment: Col¬ 
lege Learning and Democracy’s Future (National Task Force, 2012), research initiated at the Carnegie Foundation for the 
Advancement of Teaching in PEP and continued in an action project (The American Democracy Project) at the AASCU 
(Beaumont et al., 2006; Colby, Beaumont, Ehrlich, & Corngold, 2007; Goldfinger & Presley, 2010), and the Lumina Foun¬ 
dation’s Degree Qualification Profile (Adelman, Ewell, Gaston, & Schneider, 2011, 2014). Similar to the approach of 
Saltmarsh, these definitions and conceptual frameworks identify civic knowledge, skills, values, dispositions, and behav¬ 
iors as part of the learning outcomes that college graduates should possess to be prepared, knowledgeable, active, and 
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Engagement) information” (CIRCLE, 2010, p. 3). In addition to K-12 schools, these are built in higher 

education, workplaces, religious and voluntary organizations, national and community service, 
families, and neighborhoods (CIRCLE, 2010). These skills are necessary for governmental 
transparency to be meaningful, for effective participation, and for citizen collaboration. 





Degree Qualifications Profile (DQP) Civic learning Associate level: the student “describes his/her own civic and cultural background; describes 

diverse positions ... on selected democratic values or practices, and present his or her own 
position on a specific (related) problem; provides evidence of participation in a community 
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Indiana University-Purdue Civic-mindedness Civic-mindedness has these elements: “academic knowledge and technical skills, knowledge 

University Indianapolis (IUPUI) Civic responsibility of... non-profit organizations, knowledge of contemporary social issues; listening and 
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consumer-oriented action. 





Table 1 Continued. 
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students involved in political engagement projects in 21 universities’ programs (Beaumont, 2003; Beaumont, 2005; Beaumont, Colby, Ehrlich, & Torney-Purta, 2006; Colby et al., 2007). 
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perceptions (19), school 
climate (24), civic 
knowledge (6) (Flanagan 
et al„ 2007). 








Measure Source Format Delivery Length # Items Target audience Themes/topics assessed 

Civic Literacy Intercollegiate Multiple-choice Paper-and- Untimed 60 items (2007); Freshman and Measures “the top 50 themes 

Exam (2007 and Studies Institute pencil; 118 items (2008) senior students related to... ordered liberty’ 

2008 versions) 2 (ISI) National Web-based in higher in America that “capture the 
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cooperation; 6), knowledge 
(integration of learning and 
applying concepts; 3) 
(Hurtado et al., 2012b). 





International IEA & University of Multiple- Paper-and- 35 minutes 38-40 14-year olds in 28 Measures four domains of 

Association for the Maryland — many choice; pencil (multiple- multiple-choice countries civic education content: (a) 


J. Torney-Purta et al. 
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given to 17- to 19- 
year-olds only) (Amadeo, 
Torney-Purta, Lehmann, 
Husfeldt, & Nikolova, 2002; 
Torney-Purta et al., 2001). 





Measure Source Format Delivery Length # Items Target audience Themes/topics assessed 

International Civics IEA Data Multiple-choice; Paper-and- 45 minutes 80 items (74 Students in Grade 8 Measures the cognitive domains of 

and Citizenship Processing Open-ended pencil multiple-choice; if the average age knowing, reasoning, and 

Education Study Center 6 open-ended, is 13.5 and above, analyzing across four content 

(ICCS) — matrix sampled) or Grade 9 if the domains, including: civic society 


J. Torney-Purta etal. 
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understanding, and being willing 
to address social issues (Steinberg 
& Norris, 2010; Steinberg et al„ 
2011 ). 





Measure Source Format Delivery Length # Items Target audience Themes/topics assessed 

National Assessment of National Center for Multiple-choice Paper-and- 25 minutes Unknown Grade 4, Grade 8, Measures “civics knowledge, skills, and 

Educational Educational (60%); pencil and Grade 12 dispositions that are critical to the 

Progress (NAEP) Statistics Short-answer students in the responsibilities of citizenship in 


J. Torney-Purta et al. 


Assessing Civic Competency and Engagement 


Cd 


£ *£ 
£ G-, 


£ 

o 


x 


> 

U 


CL) 


>■ to 

x o a 
^ W) o 
> x '-5 

• • e* cd 
0) Cd X 

T3 ~ G 

H 3 

x .y o 

O <4H 

G X QJ 
o x 
<L> CU 3 

« <U 
T3 , a> j-, 


X 

o 

X 


O 

CL, 

g 

03 


W 

u 

£ 


a 

_ o 

CO O 

II 


cl «v: 

3 rr 


o 

CL, y 

5 I 

CL, < 


£ 2 
S o 

2 5 
b> o 

Ji 2 

+3 a 

<+-, 03 

O to 

£ I 

£ g 

.2 S 

-a 
2 o 


03 -5 

O .g 
CQ o 

C O 


G <0 

3 <y 


o 

CL, 

X 

G 


a x" 

* o 
o o 
’ M <N 

w> - 

a —j 


5 


a 


X 

w 


W Cd 

U ■ 
£ 


CL, 3 

Q W 


> 

u 


§ § 
II 

g CTN 
6 » 


<U 


bO 

.G 

g 


bo 

a 


o 

a 1 .s 

P « fl 
Cl =3 QJ 

s 3 e 

h w 51 
<U C W5 
Cl, o 


u bo 
<u d 


CL," 

o 

X - 


bO 

G 


■y x 

G c 

2 3 


” a 

Jh 

X H 
d w 
3 « 


cu 


< < 


rt 5 §3 

So B 

d 03 
<U U ' 


to d 
C/5 O 


X ft 


cd 
X) 
o , 

cd 

d 

o 


cd <l> 
<u -d 


d 

a" 


5 ° 
G <N 

<D . ~ 

SP 


bb £P 
d d 


o a 


w 


w > 
d X 
fr b> 
S 2 

^ d 

cd 


O CO 

3 Z; 


cd 

O 

x 

d 

cd 


<u .. 
bo £ 

^3 _, <u 

'd tu b b 

^ 3 d co 

dJ l« H Tl 


cd 
a> 

3 
o 

^ LO 
D-, oq 
bp I + 
d LO X 
d H rsj 


<u 

S <u 
° > 
co cd 
^ X 


.9 d 

+ d 

x o 

<N <J 


"d 

tu 

5 


S g 

n > cd 


T3 

<u 

s 


T3 

d 

cd 

a> 

d 

o , 
x x 

CL, <L> 

3t $ 

.<U 


H 


§ ^ 
^o x 




w 

X 

u 

g 

u 


d 


£ 

u 

3 

d 

o 


X 

u 


T3 

d 


d 




9 y 


'd 

<u 

a 


5 


d 


& 

CL, 


-d 

d 


O 


o 

> 

d 

CO 

3 

d 

o 


d 

<D 

a 

<L» r-H 

bp g 


cd 

£P 3 .sh 
w t9 u 




G 
<u 

d XT' x h <u 

| S ^ SP 

S 6 2 w 


16 


ETS Research Report No. RR-15-34. © 2015 Educational Testing Service 





J. Torney-Purta etal. 
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Measure Source Format Delivery Length # Items Target audience Themes/topics assessed 

Processes University of Surrey Likert-type; Paper-and- Untimed 171 Likert and 3 Young adults aged Measures political interest (3 items); 

Influencing Research Center, Multiple- pencil (typically knowledge items 16 to 26 in political attentiveness (3); past 

Democratic Martyn Barrett choice takes 60-90 Europe participation (15); effectiveness of 


J. Torney-Purta et al. 
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Measure Source Format Delivery Length # Items Target audience Themes/topics assessed 

Service Learning The Measure of Various Paper-and- Untimed Various College students, A collection of measures presented to 

(Compendium of Service Learning pencil numbers in adults correspond to a conceptualization of 

Measures) a,b (book) the 41 civic attitudes, civic skills, and civic 

different motives. Scales (described and with test 


J. Torney-Purta etal. 
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money; citizenship skills processes and 
methods (Department for Education, 
2014, Citizenship Studies: Draft GCSE 
subject content, pp. 4-10). 





Measure Source Format Delivery Length # Items Target audience Themes/topics assessed 

U.S. Naturalization U.S. Citizenship Open-ended Oral Untimed 10 items (from a Immigrants to Measures knowledge of the American 

Exam 3 and Immigration (one or pool of 100 the United government, history, and integrated 


J. Torney-Purta et al. 
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Index Project) a,b (book) campaigning), indicators of public voice 
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engaged citizens (e.g., Adelman et al., 2011, 2014; Beaumont, 2005; Hurtado et al., 2012a, 2012b; National Task Force, 
2012; Torney-Purta & Vermeer, 2006; U.S. Department of Education, 2012). Other efforts have used terms such as civic 
knowledge, literacy, and awareness; civic and democratic engagement; critical consciousness and action; social agency; altru¬ 
ism and social activism; openness to diversity and pluralistic orientations; humanitarian/civic involvement values; and civic 
communication (e.g., Hurtado & DeAngelo, 2012; Hurtado et al., 2012a, 2012b; National Task Force, 2012; Rhodes, 2010). 
Nearly all agree that civic learning is a construct of a multidimensional nature. 

The AAC&U definitions are represented prominently in Table 1. In that National Task Force (2012) report, AAC&U 
took a comprehensive view and defined the civic learning process as the educational opportunities that colleges and uni¬ 
versities offer their students to facilitate the learning of civic and democratic knowledge, skills, and dispositions through 
theory-based practice (U.S. Department of Education, 2012). The dimensions (in Table 1 under the first AAC&U entry) 
include civic literacy, civic inquiry, and civic action. Another iteration of these conceptualizations is found in AAC&U’s 
Civic Engagement VALUE (Valid Assessment of Learning in Undergraduate Education) rubric (Rhodes, 2010). These 
concepts frame the assessment of learning—diversity of communities and cultures, analysis of knowledge, civic identity 
and commitment, civic communication and skills, civic action and reflection, and civic contexts/structures — and have 
been applied in 2-year as well as 4-year institutions (see Tables 1 and 2). 

Also relevant is Hurtado et al.’s (2012a, 2012b) examination of the multidimensional nature of civic learning using 
multiple measures, utilizing the AAC&U Civic Learning Spiral framework (Musil, 2009). The authors describe this frame¬ 
work for civic learning as integrating both content and pedagogy with civic learning outcomes in institutions of higher 
education. They consider civic learning as including the knowledge, skills, values, and capacities that students ought to 
possess to be actively and purposefully engaged in society. The civic learning outcomes highlighted in their model include 
understanding of self and others, civic awareness, integration of learning, pluralistic orientation, critical consciousness 
and action, social agency, civic engagement in public forums, political engagement, and knowledge of different cultures 
and sensitivity to the issues of racism. This scope is summarized in the social change model and includes collaboration, 
common purpose, and controversy with civility (under group process values) and citizenship and change toward a better 
society (under community and societal values; Higher Education Research Institute [HERI], 1996). A meta-analysis of 
diversity-oriented programs in higher education in relation to civic outcomes found that informal interpersonal interac¬ 
tions and approaches that incorporated intergroup dialogue had special value (Bowman, 2011). 

Moving to another foundational project, PEP began in the early 2000s and involved research on 21 campuses nation¬ 
wide (see Table 1). Its influence on the field has continued with the publication of two books (Colby, Ehrlich, Beaumont, & 
Stephens, 2003; Colby et al.,2007) and articles (Beaumont, 2005; Beaumont et al.,2006), the construction of a set of assess¬ 
ment guidelines for interviews, and a survey instrument for students (see Table 2). The effort was intended to influence 
both programs and assessments. PEP has been assumed by the AASCU and resulted in a further monograph, Educating 
Students for Political Engagement: A Guide to Implementation and Assessment for Colleges and Universities (Goldfinger & 
Presley, 2010). The project concentrated more than most on activities with some political (not only civic) content. One of 
the enduring achievements of this effort is the assessment instrument produced during the research-oriented first phase of 
PEP (Beaumont, 2003; Beaumont et al., 2006). It includes assessments of knowledge or understanding, skills, identity or 
values, volunteerism, interest/motivation, efficacy, and action/involvement, as well as students’ reports of their programs 
or institutions activities (see Table 2). 

The continuing programmatic efforts of AASCU are housed in the American Democracy Project, which has several 
components, each led by specific campuses that are members of the organization. These activities are described primarily 
in publications found on websites of AASCU (2014) and AASCU/National Conference on Citizenship (NCoC; 2012), and 
include the following initiatives: PEP (described in the previous paragraph), the Civic Agency Project, and the eCitizenship 
project (see Table 1). In addition, in collaboration with the NCoC, AASCU has worked on a Campus and Community 
Civic Health mapping initiative. The Democracy Commitment at the American Association of Community Colleges is 
a partner in the American Democracy Project (Ronan, 2012). All these projects are promoting knowledge that is both 
fundamental and applied to understanding current issues, as well as enhancing skills and motivation. There has been 
recent attention to online activities in the eCitizenship Project, which focuses on the use of social networks and policy 
tools for civic purposes (AASCU, 2014) and to a Global Engagement Initiative. An overall blueprint for these activities 
can be found in Stepping Forward as Stewards of Place (AASCU, 2002), which is intended to anchor institutions in the 
communities and regions in which they are located. 
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The Degree Qualifications Profile (DQP; Adelmanet al., 2011,2014; Jankowski, Hutchings, Ewell, Kinzie, &Kuh, 2013) 
supported by the Lumina Foundation, included civic learning as a student learning outcome with competences specified 
for associate’s, bachelor’s, and master’s degree programs (see Table 1). The DQP describes civic and global learning as the 
effective preparation of students in institutions of higher education for responsible, interactive, and productive citizenship. 
In their view, students at the bachelor’s level should be able to explain diverse positions on issues, develop and justify 
positions on a public issue, collaborate with others when developing and implementing an approach to a civic issue, 
and identify significant issues affecting people throughout the world (Adelman et al., 2014, p. 19). These students should 
also be able to apply skills to contribute to the good of a democratic society (Adelman et al., 2014). The National Task 
Force report (2012) pointed to the DQP as a rich resource that exemplifies the components of civic learning outcomes 
for institutions of higher education. These components of civic learning are further embedded within the other learning 
areas of the DQP such as broad, integrative knowledge, which includes global, intercultural, and democratic civic learning, 
and also intellectual skills, which includes engagement of diverse perspectives (Adelman et al., 2011; National Task Force, 
2012). Use of the term global expands civic learning beyond the local and national levels. Additionally, possessing civic and 
global learning proficiencies prepares the student to respond to societal challenges in the micro and macro communities 
through activities that include service learning (Adelman et al., 2014). 

Selected Additional Frameworks of Civic Competency and Engagement in the United States 

In addition to the three foundational projects reviewed above, several other conceptual frameworks are found in Table 1. 
For instance, HERI describes constructs of students’ civic learning (Franke, Ruiz, Sharkness, DeAngelo, & Pryor, 2010) 
including civic awareness, which involves the comprehensive understanding of the local, national, and global communities 
and of related issues. Fikewise, HERI uses the term social agency and considers the extent to which college students value 
social and political involvement as personal goals (e.g., staying up-to-date with political news, helping others, promoting 
racial cohesiveness). HERI administers the annual College Senior Survey (CSS; see Table 2) that connects academic, civic, 
and diversity outcomes with a comprehensive set of college experiences to make inferences about civic learning in college 
(Franke et al., 2010). 

Another noteworthy framework was developed by CIRCLE, which issued and widely disseminated a paper on federal 
policy with the potential to enhance civic skills, including the ability to distinguish facts from opinions and to criti¬ 
cally analyze political information (CIRCLE, 2010). The American Association of Community Colleges has also focused 
special attention on skills of inquiry, research, participation, and persuasion (Gottlieb & Robinson, 2006), stressing the 
importance of civic competency and engagement within 2-year institutions. 

Indiana University-Purdue University Indianapolis (IUPUI), which has a Center for Service and Learning, is an 
institution where a particular segment of civic competency has been elaborated. It focuses on the integration of civic 
dimensions into knowledge obtained through study in a wide range of disciplines. The “civic-minded graduate” is some¬ 
one with an understanding of “how knowledge and skills in at least one discipline are relevant to addressing issues in 
modern society” and the “complexity of those issues” (Steinberg et al., 2011, p. 22). 

Additionally, organizations that focus on college student development, such as the National Association of Student Per¬ 
sonnel (NASPA) and American College Personnel Association (ACPA; NASPA & ACPA, 2004), include civic engagement 
as one of seven suggested student outcomes in their report, Learning Reconsidered. In addition to student leadership, they 
focus on civic values (e.g., commitment to public life) and dispositions (e.g., sense of civic responsibility). Civic engage¬ 
ment, values, skills, and dispositions are also included in the APA Guidelines for the Undergraduate Psychology Major 2.0 
(APA, 2013), endorsing ethical values that build community trust and social responsibility. In summary, a range of orga¬ 
nizations suggests that civic engagement can be fostered by general education requirements, service-learning activities, 
and social and political organizational membership. 

Similarities Between Frameworks in the United States and Europe 

Internationally, the IEA, an organization that conducts international large-scale assessments, has designed assessments of 
civic knowledge, skills, and engagement, which were administered in 1999 and 2009 (Amadeo et al., 2002; Schulz, Frail- 
lon, Ainley, Losito, & Kerr, 2008; Torney-Purta, Lehmann, Oswald, & Schulz, 2001) and are to be repeated in 2016 (see 
Table 2). In addition, in the United Kingdom, the examination for the General Certificate of Secondary Education (GSCE) 
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at age 16 (i.e., QCA, Citizenship Studies ) has developed assessments (QCA, 2007; Department for Education (UK), 2014; 
see Tables 1 and 2). A recent European Union-sponsored study that took place in eight countries (i.e., the Processes Influ¬ 
ence Democratic Ownership and Participation Study [PIDOP]; see Table 1), included a few measures of civic competency 
(e.g., political citizenship knowledge and skills) and a wide range of measures of civic engagement, including values, dis¬ 
positions, attitudes, behavioral intentions, behaviors, and aptitudes related to civics and the active citizenship capacities 
of students (see Table 2; Barrett, 2012; Barrett & Zani, 2015). Although the labeling of the components that make up civic 
competency and engagement differs somewhat across domestic and international contexts, the structure and even the 
content of the constructs is quite similar. Specifically, both groups include (a) civic or citizenship competency (i.e., knowl¬ 
edge and skills in analyzing political material) and (b) civic or citizenship engagement, including values, dispositions, 
behaviors, and self-assessed participatory skills; differences in emphasis exist between national and international entities 
(as well among organizations in each region). 

Existing Assessments and Measures of Civic-Related Constructs 

Assessments in the area of civic learning are also gaining importance (see Table 2). Measures of cognitive and attitudinal 
outcomes have existed in the United States since at least the early 1970s (when the first NAEP Civics Assessment took 
place). Further, in the early 1980s Educational Testing Service (ETS), together with the Council on Learning, conducted 
the Global Understanding Survey (Barrows, 1981), including a variety of civic-related measures. Data were collected at 
more than 180 universities in the United States. Within the last few years, it has become possible to disaggregate voting 
turnout percentages for students, and these summary figures can be reported to institutions of higher education (CIRCLE, 
2014). At the same time, assessment of students’ civic outcomes at the institutional level has become feasible. For instance, 
NSSE established a civic engagement module in the 2013 survey administration (Kinzie, McCormick, & Stevens, 2014). 
A wide range of projects in the United States and Europe at the secondary and postsecondary levels have constructed 
objective knowledge and skills items. There are also numerous self-report Likert scales for attitudes, direct assessments, 
rubrics for assessing written materials, interviews, and peer ratings. Other assessments have been designed for program 
evaluations and especially for service learning or community engagement programs. Many of these were designed for a 
specific project and are not widely transferable (according to Deardorff, Hamann, & Ishiyama, 2009). Thus, this paper 
focuses on existing measures that have been widely used and on which research has been conducted to provide a starting 
point for developing a next-generation assessment. 

Multiple Themes of Assessments 

The multidimensional nature of civic learning has led to assessments that can be classified under two major constructs; 
civic competency and civic engagement (see Table 2). Measures related to civic competency have focused on topics such as 
history, political science, economics, democracy, citizenship, civic principles, society, and government and include mea¬ 
sures such as the U.S. Naturalization Exam (U.S. Citizenship and Immigration Services [USCIS], 2011), the Civic Literacy 
Assessment (Intercollegiate Studies Institute’s National Civic Literacy Board, 2006,2007,2011), IEA Civic Education Study 
(CIVED) Test and Survey (Torney-Purta et al., 2001), and NAEP Civics (National Assessment Governing Board, 2010). 
To take one example, the conceptual framework of CIVED included four specific themes: the defining characteristics of 
democracy, citizenship rights/duties, national identity/international relations, and social cohesion/diversity. 

Other assessments related to civic competency include the measures developed by Delli Carpini and Keeter (1993), who 
used existing items from the National Election Study surveys, which were delivered in telephone interviews to develop 
and validate a 5-item knowledge index. They framed their project with a well-known definition: “The democratic citizen is 
expected to know what the issues are, what their history is, what the relevant facts are, what alternatives are proposed, what 
each party stands for, what the likely consequences are” (Berelson, Lazarsfeld & McPhee, 1954, p. 308). In a subsequent 
book titled What Americans Know about Politics and Why It Matters, Delli Carpini and Keeter (1996) analyzed results 
from phone-based surveys that had included items from the National Election Surveys, the General Social Survey, and 
an additional survey that the authors conducted. They examined data on percentage answering correctly ranging over 
several decades. The book’s appendix lists a wide variety of knowledge items. 

Fewer assessments have focused on civic-related skills, although an association of university libraries (centered at Kent 
State) has developed a Standardized Assessment of Information Literacy Skills (SAILS; Radcliff, Salem, O’Connor, & 
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Gedeon, 2007). Even though this assessment focuses on the general information literacy of students, some skills that 
are assessed directly relate to civic learning, such as the skills in evaluating sources and in recognizing social or ethical 
issues. 

Measures of civic engagement cover topics such as national identity, attitudes toward social cohesion and diversity, 
civic participation and activities, electoral and political activities, democratic values, beliefs about citizens’ efficacy, dispo¬ 
sitions, and behavioral intentions. Examples of assessments include the IEA CIVED Instrument (Schulz & Sibberns, 2004; 
Torney-Purta et al., 2001), the International Civic and Citizenship Education Study’s (ICCS) International Student Ques¬ 
tionnaire (Schulz et al., 2008), Political and Social Involvement Scale (Center of Inquiry in the Liberal Arts, 2013), School 
Citizenship and Climate Assessment (Education Commission of the States, 2006), NSSE Topical Module: Civic Engage¬ 
ment (Trustees of Indiana University, 2013), the scales from the New Civic Engagement Project (Zukin et al., 2006), and 
the Socially Responsible Leadership Scale-Revised Version II (SRLS-R2; Dugan & Komives, 2007). Additionally, two large- 
scale surveys developed by HERI (2014a, 2014b; i.e., CSS and the Diverse Learning Environments [DLE] Survey) measure 
aspects of the collegial climate, environment, and experiences and some aspects of student civic engagement along with 
sense of political agency (efficacy). 

A number of these civic engagement measures have been used in major studies. For instance, the Political and Social 
Involvement Scale and the SRLS-R2 were used in the Wabash National Study, a large-scale longitudinal study investigat¬ 
ing student learning outcomes at U.S. colleges and universities. Findings from the Wabash Study revealed that students’ 
political and social involvement increased slightly by 0.12 standard deviations (SDs) after 4 years in college, and students’ 
socially responsible leadership increased by 0.36 SDs (Blaich & Wise, 2011). Similarly, the National Civic and Political 
Health Survey (CPHS) and Flanagan, Syvertsen, and Stout’s (2007) survey measures have been utilized by CIRCLE. The 
CPHS was used to evaluate how 1,700 young people (ages 15-25) and 550 adults (age 26 and over) participated in pol¬ 
itics and community activities, as well as their attitudes toward government and current issues. Results from the 2006 
administration of the CPHS revealed that young Americans are engaged and involved in many forms of political and civic 
activity, such as voting and volunteering; however, 17% of young Americans have not participated in any political activities 
in the past 12 months. Additionally, results revealed that many Americans are misinformed and lack political knowledge 
(Lopez et al., 2006). Other large-scale studies have used instruments such as the Youth and Participatory Politics Survey, 
which was administered to over 2,500 respondents ages 15-25. This survey aims to measure “interactive, peer-based acts 
through which individuals and groups seek to exert both voice and influence on issues of political concern.” Its findings 
revealed that 41% of young people engage in at least one of these types of participatory acts, and that 84% of respondents 
are concerned about the credibility of news obtained through social media (Cohen & Kahne, 2011, p. viii). 

Item and Test Administration Format 

The existing assessments measuring civic-related constructs use various item and test administration formats. A majority 
of the assessments employ selected-response items. Multiple-choice items are used in many of the assessments measuring 
civic competency, while Likert-type items are primarily used for measures of civic engagement. Yes/no items typically 
ask about the involvement of a respondent in various activities, such as whether a person voted in an election or signed 
a petition. Likert-type self-report items focus on respondents’ levels of agreement, perceived importance, frequency of 
participation in certain activities (e.g., voting, petitions, political meetings, volunteering in the community), or satisfaction 
from participation in those activities. The Defining Issues Test-2 presents problem-based scenarios (several with political 
content) and asks students to rank (rather than rate) a series of issues that might be relevant to making a particular decision 
(Rest & Narvaez, 1998; Thoma & Dong, 2014). The IUPUI Center’s measure of the Civic-Minded Graduate (Steinberg 
et al., 2011) also includes a problem-solving scenario along with Likert ratings and a written narrative from which both 
knowledge and engagement are assessed. 

Open-ended items, such as short-answer and essay items, are less common among civic competency and engagement 
assessments but are used on the ICCS’s International Cognitive Test (Schulz et al., 2008), NAEP Civics (National Assess¬ 
ment Governing Board, 2010), NSSE Topical Module: Civic Engagement (Trustees of Indiana University, 2013), and the 
IUPUI Center’s assessment of the civic-minded graduate (Steinberg et al., 2011). Open-ended items can also be found on 
the United Kingdom’s GCSE examination in Citizenship Studies (Department for Education (UK), 2014). The first part 
includes written short answers and an essay, while a second part is comprised of a controlled project assessment (completed 
by the examinee with teacher oversight; Brett, 2002). Both parts of the examination deal with applying cognitive skills as 
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well as factual or conceptual learning. Test administration also varies, with most assessments using a paper-and-pencil or 
web-based format and others using an oral format. For instance, the U.S. Naturalization Exam (USCIS, 2011) uses open- 
ended items with one- to two-word answers given orally. Other assessments have used an oral format through phone-based 
interviews such as the 2008 version of the Civic Literacy Assessment (Intercollegiate Studies Institute’s National Civic Lit¬ 
eracy Board, 2011) and theCPHS (Lopez et al.,2006). For several decades, public opinion organizations have administered 
knowledge items to adults in phone interviews; the focus is generally on current events knowledge (usually about national 
and foreign policy issues). 

Test and Scale Reliability 

Reliability estimates range from .00 to 1.00, with .00 indicating that all of the variance in the score is due to measurement 
error and 1.00 indicating perfect reliability with no measurement error. Whether the internal reliability for an assessment 
is acceptable or not hinges on the testing purpose and the context of score use (Haertel, 2006). Typically higher reliabilities 
are required when higher stakes are involved in decision making based on the test scores (American Educational Research 
Association [AERA], APA, & National Council on Measurement in Education, 2014). For instance, assessments that are 
used for admission to an institution of higher education would require higher levels of reliability than assessments to 
compare groups of individuals. Frisbie (1988) noted that “experts in educational measurement have agreed informally 
that the reliability coefficient should be at least .85 if the scores will be used to make decisions about individuals and if 
the scores are the only available useful information” (p. 29). However, “the need for precision [i.e., reliability] increases as 
the consequences of decisions and interpretations grow in importance” (AERA et al., 2014, p. 33), meaning that the level 
of satisfactory reliability is dependent on the stakes of the assessment. A number of variables can impact an assessment’s 
reliability, such as test length, item types, item quality, the group of examinees, and the conditions of test administration 
such as instructions and time limits (Traub & Rowley, 1991). 

Given the multifaceted nature of civic-related constructs, many assessments include subscales and report subscores. The 
Standards for Educational and Psychological Testing (AERAet al.,2014) states that the decision to provide subscores should 
be made carefully, and that both the “distinctiveness and reliability of separate scores should be demonstrated” before 
reporting any subscores (p. 27). Many existing civic assessments have reported subscores with reliability estimates above 
.80 (e.g., ICCS’s International Student Questionnaire, DLE survey, SRLS-R2, IEA CIVED Instrument, and the Political 
and Social Involvement Scale) despite the fact that some scales have relatively a small number of items. However, some 
existing measures have some subscores that have not met the criteria for satisfactory reliability. For instance, on the PEP 
instrument, four of the 30 scales showed lower internal consistency ranging from .65 to .69 (2-4 items; Colby et al., 
2007). Similarly, for CIRCLE’S scales, Flanagan et al. (2007) reported that the large majority yielded reliability estimates 
greater than .80; however, many of the subscales reported reliability estimates above .70, and a few subscales had reliability 
estimates between .65 and .70. This finding was likely related to only 3 or 4 items in those subscales. Depending on the 
stakes of these assessments, even these reliabilities could be considered adequate. 

Although the subscores reported by many existing measures have demonstrated satisfactory reliabilities, there has been 
little evidence demonstrating subscore distinctiveness. Torney-Purta et al. (2001) evaluated the IEA CIVED instrument 
using confirmatory factor analysis to determine the appropriateness of using two subscores (i.e., knowledge of content 
and skills in interpretation of civic-related material). Although the subscores were highly correlated (r = .91), the two- 
dimensional model showed a slightly better fit. The authors argued that it was valuable to report these subscores because 
it led to a “better understanding of the relative strengths and weaknesses of civic knowledge as developed in participating 
countries” (Torney-Purta et al., 2001, p. 63). Furthermore, in a secondary analysis of CIVED data from the United States 
using cognitive diagnostic modeling, Zhang, Torney-Purta, and Barber (2012) found differences between those respon¬ 
dents who excelled on civic skills and those who excelled on conceptual knowledge of civics in the extent to which they 
had received conceptually based teaching in their social studies classes. 

For open-ended items, reliability is typically reported in the form of interrater reliability to evaluate the consistency 
between scores given by multiple raters. On NAEP Civics, interrater reliability estimates are computed by using the percent 
of exact agreement of two raters scoring responses to an open-ended item. These ranged from 77% to 94% for Grade 4 
responses, 68% to 94% for Grade 8, and 66% to 97% for Grade 12 (U.S. Department of Education, 2011). Additionally, 
Winke (2011) examined the reliability of the U.S. Naturalization Exam, which uses oral, open-ended test items. This study 
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found reliability estimates around .71. The author also noted that 14 of the 100 test items were unreliable and recommended 
that they be removed from the assessment pool (Winke, 2011). 

Validity Evidence 

Relatively limited validity evidence is reported in the literature for the existing assessments measuring civic-related con¬ 
structs. Delli Carpini and Keeter (1993) addressed validation using expert judgments and correlation analyses. Other 
validation studies have focused on evidence based on internal structure (i.e., dimensionality) as discussed in the Stan¬ 
dards for Educational and Psychological Testing (AERA et al., 2014). This type of validity evidence indicates whether the 
associations among test items correspond to one or several intended constructs (or dimensions) of the assessment (AERA 
et al., 2014). Confirmatory factor analysis (CFA) is one of the most frequently used methods to evaluate the internal struc¬ 
ture of an assessment. CFA compares the hypothesized and observed test structures by examining how the test items relate 
to the intended theoretical constructs of the assessment (Brown, 2006; Rios & Wells, 2014). Indices of model fit are used 
to determine whether the assessment is measuring what it is intended to measure based on the structural relationship 
between the test items and the construct(s). 

Hurtado, Arellano, Cuellar, and Guillermo-Wann (2011) used CFA to evaluate the internal structure across the three 
subscales of the DEE survey that targeted civic competency and engagement, including pluralistic orientation, civic action, 
and social action engagement. Results of model fit indices suggested that the items across each of the three subscales were 
adequately measuring the intended constructs. Similar analyses were conducted by Lott and Eagan (2011) to evaluate the 
internal structure of the civic values domain for the CSS. Using CFA, the authors confirmed that the eight items adequately 
measured the subdomain of interest. 

Winke (2011) investigated the validity of the U.S. Naturalization Exam, an open-ended orally administered assessment. 
To administer the assessment, a USCIS officer selects 10 test items from a pool of 100 test items. The author found that 
this pool of 100 test items could be separated into approximately five distinct test forms of citizenship knowledge based 
on item difficulty. There is no documentation that indicates how or whether USCIS officers choose a selection of items 
of equal difficulty to administer to each applicant. If not, this would make the assessment unfair to some test takers. The 
author also found that of the 100 items, 23 possessed differential item functioning (DIF) with 10 items being easier for 
U.S. citizens and 13 items being easier for noncitizens. 

Assessments that reported relevant evidence have in general demonstrated adequate construct validity. However, more 
evidence is needed to support the intended uses of test scores (AERA et al., 2014; Kane, 2013). For example, since many of 
the assessments report subscale scores, it is important to examine the multidimensionality of the underlying constructs. 
Furthermore, as previous research has shown differences in the level of civic competency and engagement by ethnicity 
and gender (Lott, 2013), future research should evaluate the extent to which these civic-related constructs are measured 
similarly across demographic groups (see the next section). 

Challenges in Designing a Civic Competency and Engagement Assessment 

Common challenges exist when developing assessments, such as appropriately addressing content, task design, and scor¬ 
ing concerns, as well as adequately meeting validity and reliability requirements (Downing & Haladyna, 2006; Haladyna & 
Rodriguez, 2013). However, unique challenges specific to the measurement of civic competency and engagement can also 
be expected. For instance, respondents may have a desire to appear more civically engaged on a self-report measure of civic 
engagement (potentially resulting in distortion of responses). Second, there is the issue of reliability of subscores for mul¬ 
tidimensional themes within civic knowledge and engagement. Next, we need to consider the setting or contextualization 
of the construct being measured, and finally, we need to consider subgroup differences. 

Inauthentic Responding in Measures of Civic Engagement 

As self-reports are commonly used in assessments for civic engagement, the genuineness of these responses may be a 
concern, especially if high stakes are attached to the assessment. In fact, the tendency for individuals to report themselves 
as having socially desirable or valuable characteristics has long been a concern with self-report measures (Spencer, 1938). 
In other words, there appears to be a tendency for a respondent to either consciously or subconsciously provide inaccurate 
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responses to make himself or herself appear socially involved. In a review of 51 experimental studies, score differences due 
to this type of response (sometimes called faking) in Likert-type items on personality inventories ranged in absolute value 
from 0.48 to 3.34 SDs (Viswesvaran & Ones, 1999). As a result, there is a need to explore possible solutions to improve 
score-based inferences from self-ratings of civic engagement. 

Researchers have experimented with innovative ways to assess constructs that typically rely on self-reports. These meth¬ 
ods include the use of warnings and alternative item types (i.e., non-Likert-type items) to either identify or decrease the 
likelihood of the tendency to give socially desirable responses. Warnings have been found to have only a small impact on 
mitigating inauthentic responses when looking at the standardized mean difference (Cohen’s d = 0.23; Dwight & Dono¬ 
van, 2003). This has led researchers to recommend two possibilities: internal and external techniques. 

The first approach consists of including external measures (i.e., social desirability or bogus items) in an assessment. For 
example, researchers have taken previously developed social desirability Likert-type items (e.g., from Crowne & Marlowe, 
1960) and inserted them into an unrelated assessment. In contrast, the bogus statement approach involves developing 
items that appear to be related to the construct, trait, skill, or task of interest, but where the objects or situations described 
in the items do not exist. Examples developed by Dwight and Donovan (2003) include: “How often do you access online 
chat rooms for the International Student Excellence Group?” or “How often do you utilize murray-web system to locate 
unpublished research articles?” where neither the International Student Excellence Group nor the murray-web system 
exists (p. 10). It is assumed that endorsing these items containing bogus statements indicates that the examinee has a 
tendency to provide untruthful responses. 

The assumption underlying the use of external measures is that if respondents have high endorsement on both the 
external items and assessment of interest, their high score is likely contaminated with an attempt to “look good.” However, 
social desirability items appear to be error-ridden indicators of inauthentic responding (Burns & Christiansen, 2011; Tett 
& Christiansen, 2007), whereas bogus items have been shown to have other difficulties (Dwight & Donovan, 2003). As a 
result of the limitations associated with the inclusion of external measures, there has been interest in other methods for 
reducing the extent to which respondents report attitudes or behaviors that are uncharacteristic for them (i.e., chosen in 
an attempt to portray themselves in a positive light). 

The second approach uses internal methods to attempt to curtail respondents’ attempts to make themselves appear 
socially adept by designing items in new formats. There have been two major advances: forced-choice items and situational 
judgment items. Forced-choice items require the respondent to choose one of two (or more) options that appear equally 
desirable with each option representing a different trait (Christiansen, Burns, & Montgomery, 2005). An example of a 
forced-choice item is demonstrated by Meade (2004, p. 535) is presented below: 

Choose one of the following 

Item 1:1 am the life of the party (measures extraversion) 

Item 2:1 follow a schedule (measures conscientiousness) 

In the example, both response options are assumed to be of approximately equal social desirability. However, each 
response option represents a distinct construct (i.e., extraversion and conscientiousness, respectively). A disadvantage is 
that a relatively large number of these paired items is typically required to obtain sufficient information on the examinee’s 
standing on a construct. Furthermore, a number of psychometric scoring concerns are related to the ipsative nature (i.e., 
all response options sum to the same total) of this item type. 

In contrast, in situational judgment items, a respondent is presented with a task-related situation, which can be written, 
video-based, or multimedia in format, and is asked to choose an appropriate response from a list of alternatives. The item 
does not require the respondent to report his or her behavior, but rather it can be viewed as a situational interview (Lievens, 
Peeters, & Schollaert, 2008). Peeters and Lievens (2005) developed the following situational judgment item for assessing 
college student success through various constructs such as student work habits: 

You have so many assignments to complete and so much studying to accomplish, you feel you will never get caught 

up or accomplish anything. You are truly overwhelmed. What would you do? 

a Prioritize your activities, enumerate the steps to be accomplished for each activity, and systematically go through 
your work, (correct response) 
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b Decide what you can accomplish reasonably and focus on getting that work done, and let [leave] the rest of the work 
unfinished. 

C Talk to your professors, explaining your situation, and ask for extensions on the due dates. 

d Take a break for a day and go out with your friends, then go back to working hard again, (p. 84) 

Situational judgment items often present the examinee with a number of appealing response options; however, there are 
a number of different procedures for scoring that include (a) the test author or developer determining the correct answer, 
(b) a group of experts deciding on the best or most correct answer, (c) allocating a score to each option based on the 
percentage of people choosing that option, and (d) selecting the best response based on the strongest predictive validity to 
a criterion of interest (e.g., job or task performance; Strahan, Fogarty, & Machin, 2005). The latter scoring option is akin 
to that used in selection tests for employment (Arthur, Glaze, Jarrett, White, Schurig, & Taylor, 2014; Campion, Ployhart, 
& MacKenzie, 2014; Whetzel & McDaniel, 2009). 

Although both of these item types show promise for reducing the tendency toward inaccurate reporting of one’s 
socially desirable attitudes or behavior, greater emphasis has been placed on forced-choice items. For example, in com¬ 
paring forced-choice and Likert-type items, Martin, Bowen, and Hunt (2002) found significantly higher mean scores 
attributable to creating a socially desirable impression for Likert-type items, but no such trend was observed for forced- 
choice items. Similarly, Jackson, Wroblewski, and Ashton (2000) found that faking on an employment test with Likert- 
type items resulted in a positive mean difference of approximately 1 SD. The use of forced-choice items reduced this 
to 0.32 SDs. These findings suggest that forced-choice items have the potential to mitigate inauthentic responding to 
self-report instruments and could be incorporated in civic engagement assessments to strengthen the validity of score- 
based inferences. However, relying primarily on forced-choice items would result in the need for an increased number 
of items. 

Establishing Reliable and Distinct Subscale Scores 

Frameworks and existing assessments of civic learning show that the construct is multidimensional, which necessitates 
the consideration of subscores. As we discussed in an earlier section (see “Test and Scale Reliability”), reporting subscores 
requires that the scores be reliable and distinctive from each other. In the case of this proposed framework, we plan to 
consider two subscores for civic competency and civic engagement. 

Although subscores have the advantage of providing information about an examinee’s strengths and weaknesses 
(Traub & Rowley, 1991), evidence needs to be collected to support the specifics of subscore uses (Kane, 2006). Inaccurate 
information provided through subscores can misinform score users when high-stakes decisions are made based on 
those scores (Sinharay, Haberman, & Puhan, 2007). A number of methods can be used to evaluate the appropriateness 
of subscores by evaluating the assessment dimensionality. Common methods include factor analysis or multidimen¬ 
sional item response theory (MIRT; Sinharay, Puhan, & Haberman, 2011). Additionally, research has demonstrated 
alternative approaches for reporting subscores such as reporting weighted averages (e.g., Sinharay, 2010) or augmented 
subscores (i.e., creating subscores by borrowing information from other portions of the test such as other sets of items; 
Wainer, Sheehan, & Wang, 1998). That said, although these alternative reporting approaches have the potential to 
provide accurate diagnostic information, they may be difficult to explain to the general public or test users (Sinharay 
et al., 2011). 

In addition to evaluating the reliability and distinctiveness of subscores, researchers have also argued that it is important 
to determine whether subscores have added value over total scores (e.g., Sinharay, 2013; Sinharay et al., 2007; Sinharay 
et al., 2011), meaning the susbcore should not be too highly correlated with the total score. Strong relationships between 
the subscore and total score would suggest that the two scores are measuring the same underlying skill and that the 
subscore does not provide any additional information apart from the total score (Sinharay et al., 2011). Sinharay (2010) 
conducted a simulation study and found that for subscores to have added value, they should be based on roughly 20 items 
and should be sufficiently distinct from each other, with correlations less than .85. 

Context and Its Impact on Assessments of Civic Competency and Engagement 

Various issues have been raised regarding the context or setting focus in both the educational process and the assess¬ 
ment of civic competency and engagement. This includes discussions about the contextualization of the constructs 
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(i.e., campus, local community, workplace, national, or global focus) and the impact of the focus chosen on the 
assessment of students from diverse social, cultural, and nationality backgrounds (Davies, 2006; Haste, 2010; Kerr 
& Cleaver, 2004). 

First, there are differences among the institutions where assessments might be employed (Ostrander, 2004). Two-year 
colleges often have many students who are part-time, commuters, or studying primarily online, providing a different con¬ 
text than most 4-year or residential campuses. There may be a historic commitment to public benefit or humanitarian 
goals, such as that found in some land-grant or religiously based higher education institutions. The political jurisdictions 
in which institutions are located vary a great deal—providing a context in which college students are welcomed or dis¬ 
couraged to participate politically, a context with more or less politically competitive elections, or a context where there 
are stronger or weaker civil society organizations. The economic conditions in neighborhoods surrounding some institu¬ 
tions may give urgency to projects in the local community. Some of these challenges are important to discuss, although 
there is not sufficient evidence to deal with many of them. 

Second, terms such as globalization, multiculturalism, cosmopolitanism, and pluralistic orientation, among others, are 
used in the literature to highlight another focus that a civically competent and engaged college student should develop. 
Burgeoning social media outlets have provided a platform for citizens around the globe to plead, organize, and fight for 
freedom from oppression as well as to practice effective political consumerism (Anduiza, Jensen, & Jorba, 2012; Banaji 
& Buckingham, 2013; Barrett & Zani, 2015; Kahne, Lee, & Feezell, 2013). A relatively recent focus on citizenship with a 
global perspective is seen by many as the vanguard of both education and the assessment of civic competency or engage¬ 
ment and is valued by key stakeholders and researchers, including the U.S. Department of Education (National Task Force, 
2012) as well as more broadly (Osier & Starkey, 2006). Many employers also believe that awareness of international pro¬ 
cesses and cultural practices is an essential component of preparation for success in the workplace (e.g.. Hart Research 
Associates, 2015). As a result, context, especially globalization, should be considered when developing an assessment of 
civic competency and engagement. 

Fairness With Regard to Subgroups of Respondents 

Another challenge when developing an assessment of civic competency and engagement is possible subgroup differences. 
Haste (2010) delineated some contested education and assessment practices to consider when measuring the civic com¬ 
petency and engagement of students who differ in ethnicity or cultural background, with a special focus on international 
or immigrant students. For instance, differing views of government social welfare programs exist among individuals from 
the United States and from Europe. Civic engagement norms also vary. For example, there is evidence that purposeful 
volunteering, often cited as a desirable civic engagement behavior, is valued differently by individuals from the United 
States than by individuals from former communist nations. Lastly, it is essential to take into consideration that the demo¬ 
cratic lived experiences of individuals vary between countries because of distinctive histories of democracy (Haste, 2010). 
Furthermore, immigrants are likely to be especially interested in political issues that have a potential impact on their 
countries of origin. 

On the topic of gender, Delli Carpini and Keeter (1996) found that adult males excelled on political and civic knowledge 
items when the topics dealt with war and the exercise of political power (which predominated in most surveys of adults that 
they examined). Females performed better when the political topics related to social welfare policy or education. Dolan 
(2011) obtained similar results showing males outperforming females on political knowledge; however, on questions about 
political knowledge focusing on the status of women in American politics, the gender disadvantage disappeared. That 
said, Torney-Purta et al. (2001) found that only one country out of 28 in the IEA CIVED study showed significant gender 
differences in knowledge scores. Under civic engagement, some argue that volunteering is more likely to be engaged in by 
females (e.g.. Bureau of Labor Statistics, 2015; Einolf, 2011). In this area, males maybe disadvantaged. Reason and Hemer 
(2015) in their literature review concluded that “women seem to have higher scores [on civic engagement], but that isn’t 
universal” (p. 30). To the extent possible, the profile of issues and topics should be balanced in relation to both genders in 
the assessment. 

Racial differences should also be considered. Results on Grade 12 NAEP Civics showed that from 1998 to 2010 the 
performance gap between White and Hispanic students has narrowed but has stayed the same between White and Black 
students (National Center for Educational Statistics, 2011). White students outperformed Black, Hispanic, and American 
Indian/Alaskan Native peers, which is consistent with prior research indicating that White students are more likely to 
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have opportunities to engage in various civic activities that are considered interactive, such as debates, mock trials, and 
discussions of social issues, when compared to Hispanic/Latino and African American students (Kahne & Middaugh, 
2008; Kawashima-Ginsberg, 2013). Several projects (including some mentioned in the text and in Table 1) have given 
attention to racial diversity (Cohen, 2010; HERI, 2014b). Reason and Hemer (2015) found mixed results by racial group 
in their review. 

In addition to the consideration of international/ethnic, gender, and racial differences, it is also important to exam¬ 
ine how the courses a student has taken or a college major could impact performance on an assessment of civic capacity 
and learning. Delli Carpini and Keeter (1996) found that adult respondents who reported having taken civic education 
classes in high schools demonstrated more knowledge about civic topics typically included in those classes, while adults 
who reported regular reading of the newspaper had more knowledge on the topics of political parties and leaders. Similar 
results have been found on Grade 12 NAEP Civics, with students who reported studying civics or government in high 
school scoring higher than those who did not (National Center for Educational Statistics, 2011). These results suggest 
that civic-related courses have the potential to impact civic competency and engagement. As another example, students 
in a political science major may be more likely to participate in institutional activities related to civic engagement than 
students majoring in English. Factoring in these considerations, assessments should be broad enough to integrate disci¬ 
plinary studies and also have crosscutting proficiencies that college graduates need for continued learning in complex and 
changing environments (Adelman et al., 2014). 

A Proposed Assessment Framework for a Next-Generation Civic Competency and Engagement 

Assessment 

Based on a review and synthesis of the existing frameworks, definitions, and assessments, we propose an assessment 
framework based on the higher-level construct of civic learning containing two domains: civic competency and civic 
engagement (see Table 3). 

Civic Competency 

In this framework, civic competency is composed of three components: (a) civic knowledge (conceptual as well as fac¬ 
tual knowledge), (b) analytic skills, and (c) participatory or involvement skills. Many of the frameworks and assessments 
reviewed include this competency component in some form (see Tables 1 and 2). Civic competency is a critical compo¬ 
nent because both institutions of higher education and employers expect the acquisition of knowledge and skills to be 
an important aspect of civic-related learning in higher education. Materials covered during instruction in the social sci¬ 
ences (e.g., introductory courses in political science, economics, sociology, and history) transmit part of the content to be 
assessed under civic competency, but other aspects of the college experience also contribute (e.g., leadership experience 
in campus organizations, experience in dealing with complex social issues during community service, and participation 
in online communications). 

Civic Knowledge 

Possessing knowledge is important in itself as a part of civic competency. It also allows individuals to understand current 
events (particularly as they are presented in print or online) and make reasoned judgments about their own participation 
in political discussion and actions on the campus, in the community, or online (dealing with national and international 
events). It is hard to imagine an adult feeling efficacious or prepared to take political action in the absence of civic 
knowledge—conceptual as well as factual, historical as well as contemporary. In short, we posit that a minimum level 
of knowledge is essential for civic competency. 

To give some examples relevant for students in the United States, these are components of the knowledge portion of 
civic competency: 

• knowledge about fundamental concepts and principles (for example, the rule of law and civil rights) and the history 
of democratic institutions (especially in the United States); 

• knowledge about political institutions as well as major political and social issues; also the complexity of social prob¬ 
lems and their solution; 
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involving diverse communities) 

^Evaluate strengths and weaknesses of potential approaches to civic and political prob¬ 
lems and be reflective about decisions and actions 

=Be reflective about the potentials and challenges of social media in politics 





J. Torney-Purta etal. 


Assessing Civic Competency and Engagement 


P 

.p 

p 

o 

U 


3 


Ph 

a 

s 


CJ 

CO 

<U 

TO 

CJ 

CJ 

2 

»H 

CJ 

p 

tj 

cu 

0 

a 

0 

CO 

Sh 

Ph 

Sh 

<u 

O 

<U 

'P 

CO 

Ph 

'P 


'P 

P 

p 

QJ 

TO 

0 

TO 

2 

2 

*H 

bO 

CJ 

Sh 

Sh 

2 

O 

Td 



P 

CU 

cu 

TO 

<u 

Sh 

O 

CJ 

_P 

2 h 

co 

2 


0> 

a 


2^ 

Cl, 

£' 
TO 

o 

S' 


bO 

a 


I p 

^ Ph 
c3 

£ § 
JJ rt 

*£ 

o - 


O 

a, 


§■ 


3 2 

P-i 2 


’<J 4P CU 

O ^ P 

<U P 
* 2 22 


Td 

a -5 

TO P-i 

TO 

s & 

•p bo 

2 g 

2 2 
Ph ^ 

o T) 

bo- a 

a ^ 

g p 
J2 03 

2 r r 
u 8 


to P 

Sh O 

CU 1/3 


o 

bO 

.P 

P 

TO 


Td 

a 


p 

,p 

Td 

p 

TO 


bo 

P 

3 


! i 

i n 

£ ■§ 
P rt 


Td -3 

P to 

TO ;jh 
TO 2 

bP p 

P Sh 

p P 
5 o 


a 

o 

u 

Ph 

a 


bo 

P 


Td 

P 

TO 


P 




a 

p 

bD 

H 

TO 

bO 

.P 

TO 

3 


p 

o 

cj 

p 

p 

TO 


a 

o 

CJ 

p 

o 

Td 

p 


-p 

p 

Ph 

p 

p 

TO 

bD 

P 

j> 

o 


4b 

o 


bO 

p 

a ; 


p 


o 

Ph 

Sh 

P 

Ph 


Ph 

P 

_TO 

’Hh 

o 


-P 

p 

Ph 


a g 


a 


p p 

P p 
‘bo to 

TO ^ 

.a g 

co .P 
p 2 

o 2 

Ph ^ 


'to *±3 

CJ CU 

2 'p 

'o a 

Ph 

p 


TO 


a> 


p w 

a -p 

0 TO 
^ § 
TOd p 

p <u 
to bo 

<J -P 

:s | 

^ I 


b Td 


Td d 
>h .p 
TO Sh 
Td Ph 

TO ^ 
5 P 


o 

-m' 

2 


■TO <U P 
•t3 Sh O 

8 

P 

44 

TO 

Sh 

CU 

^ Ph CO 

43 

( >^ 
'H—1 

£ 

O 

2 

a> 

Td 

p 

p 



43 

2^ 

0 

a c s 

P 


Ph 


a, 1- 

< 2 < 

cu 

Td 

* 

2 

? 

Td 

p 

TO 


bo 

Td 

P Ph 
P O 


bo . 


8 

3 ^ 


Td 2 

p ^ 

* s 

o S 

a £ 
.& 3 

a J 

TO 


O 

-m' 

P 

I 

a 

o 


M “ 2> 

8 £ % 
p O <u 


o 

bo 

P 

2 

p 

TO 


Td 

a 


o 

4 b 

TO 


Td 

P 

TO 


P 

Td 

;g 

2 

.p 

43 

! I 

a, 4 

•tf <U 

U 

2 P 
TO TO 


P 

Td 

o 


bo 

P 

2 


bo 

P 

2 

:p 

p 

4b 

Ph 

P 

O 

Sh 

bO 

.P 

P 

O 


Ph 

o 

o 

CJ 

bO 

.P 

’p 

P 

TO 

43 

P 


P 
O 

p g 


TOd 

o 

a 


*±3 


o 

o 

4b 


I 

E 


Td 

Td 

P 

TO 

bO 

p 


2 2 
4^ g 

^ cj 
Ph (j 

13 P 
O P 


Ph 

P 

o 

bo 

TOd 

P 


bo TO 
TOd 
P 

TO 


Ph (U 

2 -$ 

§ ^ 

Td .a 

TO -Td 

TO | 

TO TO , 


3 a 

Ph g 
O O 
+-* <u 

^ Vh 

O P 

E 2 

o X5 


o 

Ph 

.§ 2 
iu P3 

a § 


a ■£: .9 

b/5 0> to 
»H TO .4 

O O Q 


o 


'J 

CtH 

CO 


0 

<U 

=b 

O 

p 

'J 


Ph 

0 

b> 

Td 

.CO 


Td 

CJ 

2 


P 

TO 

_p 

0 

C+H 

CO 

bD 

C 3 

a 

0 

bO 

2 

0 

CJ 

2 


1 

a 

TO 

43 


43 

2 p 

-4 CU 

.a a 

^ 8 
TO 

P ^ 

I « 

TOd p 

a 2 

TO nj 


O 

Ph 


a l 

TO co 

§ 3 

2 a 


1 o 

•P CJ 
^ £ 


TO 

Ph 


a 8 

Ph P 
<U 'P 

P P 
to 


. TO 


g> P 

2 p 

<U o 
-p p 

.5 o 


p p 

p a 


a> 


< ^ 


TO TO y 
Sh d 2 h 

Ph 'Td g 

2 £ P 
co "d 

<5 2 

P C-M P 
W) o p 
g cj 52 

4b co "P 

P Sd 

p ^ 

CO Ph 


'J 

TO 

Ph 

n 03 
p P 


'p 

p 

<u 


TO 2 
P s 

S 2 


TO t3 
bo 2 

53 2 
<U Ph 


0 

cu 

P 

Ph 

TO 

Td 

d 

p 

'J 

p 

<u 

2 

0 

bo . 

TO 

44 

.P _j 


TO 


2 

Sh 

3 • 

<u 

<U 


Td 

Td 

p ! 

p 

p 

O ! 

p 

p 


0 

Sh 

p ! 


O 

TO i 


2 'p 

TO P 


a 2 
s> 2 

TO P 
W5 o 

2 
0> TO 

# p 2 

’> o 

u 2 




32 


ETS Research Report No. RR-15-34. © 2015 Educational Testing Service 






Construct domain Definition Examples of assessment topics (based on themes from Tables 1 and 2) 


J. Torney-Purta etal. 


Assessing Civic Competency and Engagement 


Td 


£ , 
d 
Td 

U 

o 


Td 

d 


CL, 

Td 

Jh 

w 

I 


Td 

d 


Td 

3 


> £ 

•a s 

o 0 

d U 


< 


£ 

§ 

o 

Oh 


ctf 

Td 

d 

d 

a> 

E 

a> 

DC 

03 

DC 

a 


DC 

# d 

’3 

I 


o3 

o 

"dc 


Td 

a 


d 


nd 

o 

o 

DC 


_o 

d 


x> 

o 

'DC 
Td 
-d 3 
o * 

S> "s 

.« o 

1 1 


DC O' 

a 3 


-d ’ 

o 

o ■ 

£ 

'3 

d , 

3 - 


^ S 

■3 P 


<u 

d 

03 


•d Td 

is 

S 3 P 


£-2 
I * 


Td 

d 


>- o 


DC 1/3 
d jh 
<u o 

13 '3 

- § 


d 

o ■ 

<-M ’ 

O . 

Td 

d 

d 

o 

DC 

^ • 

o3 


d 

DC 

d 


d ■ > 

<u ^ 

CO K> 


& U 


O 2 

3 03 

CL) <*> 
Td 03 
<+h m 

o rS 


.d 

o 

d 


d ^3 


Td 

d 


o 


CQ 


d Td 
o o3 . 

•jd o 

.Th 5h 

t> 'f 3 
cc £ 
a> o 
"3 d 

d 

.d >h 
DC O 
'm c/3 

O P 


-O 

d 

d 


ctf 

Td 


Td 

d 

03 

DC 

d 

£ 


.§> a 

B “ 

d M 


DC 
d • 


a & 

a o 

bS &> ’ 
^ g 

o3 *43 


3 

d 

1 - 


-d 

d 

d 

Td 

d 


Td 

d 


d o > 


<u 

-d 

£ 

3 

-d 

d 

d 

q 

CL, 

CU 

o3 


o 

g 

DC 

o 


-d 

(U 

-d 

'% 

d 

o3 

cm 

o 

o 

cm 

d 

:g 

■9 

d 

’3 

‘d 

o 

-d 

'oj 

s 

jd 

O 

CU 

C/3 

d 

O 

<U 

C/3 

d 

3 

DC 

o 

u 

'—( 

<u 

CJ 

d 

o 

d 

o3 

<u 

1 

C/3 

d 

Td 

_d 

<u 


(U <u 

13 


d 

jd 

> 

Td 

d 

03 

d 

o 


*g .§ 
d U 


Td 

d 

03 

DC 

t d 

d 1 

3 

d 

o 


o3 03 

d ‘|j 

'o cd 
•jd o) 
h d 

03 CU 

d -d 


O 

d 

d 

£ 

o 


o3 

d 

O 


Td 

d 

o3 


£ 

-d 


DC 

d 


* * 
.t; O 
d c 


DC 

d 


£ & 

&-S 

Td ^ 
d « 

^ ni 


> 

U 


o3 

Td 

d 

03 

d 

o 


d d 


o 

o 

Cfl 

d 

'DC 

a 

M 

o 

03 

d 

CU 

_o 

d 



o3 

d 


ETS Research Report No. RR-15-34. © 2015 Educational Testing Service 


33 






J. Torney-Purta et al. 


Assessing Civic Competency and Engagement 


• knowledge of the legal aspects of citizenship, the right to vote, and what political representation entails; and 

• knowledge of how practices and events in the local community or the nation relate to a global perspective. 

Further details about these aspects of knowledge can be found in Delli Carpini and Keeter (1996), National Assessment 
Governing Board (2010), Intercollegiate Studies Institute’s National Civic Literacy Board (2011), National Task Force 
(2012), and Torney-Purta et al. (2001). 

The knowledge that is assessed should be nontechnical and accessible to students from a range of majors (not limited 
to political science, history, economics, or sociology). In many cases, students’ civic knowledge will have been acquired in 
general studies courses in college (or in high school courses), in cocurricular activities (including service-learning expe¬ 
riences), through reading of national and international news (online or in print), or during discussion with others who 
are members of the faculty, their peer groups, community groups, their families, or online (e.g., in blogs or tweets). Some 
believe that this knowledge should focus on the history of the U.S. political institutions and the Constitution (Intercolle¬ 
giate Studies Institute’s National Civic Literacy Board, 2011), as well as the ability to comprehend terms relevant to national 
political institutions and their processes, for example, caucus, checks and balances, or due process of law (Hirsch, Kett, & 
Trefil, 2002). Others such as Hatcher (2011), believe that students’ knowledge should also include information about the 
distribution of power in society and the accomplishments of major social movements that took action on contested polit¬ 
ical issues. The sample topics listed under civic knowledge in the assessment framework found in Table 3 were distilled 
from the conceptual frameworks in Table 1 and the measures in Table 2. 

Analytic Skills 

The analytic skills component of civic competency focuses on the ability to systematically analyze written material from 
charts and graphic material, texts (including but not limited to those that might appear in the media), or political cartoons. 
The National Task Force (2012) and the VALUE rubrics (Rhodes, 2010) considered the importance of critical analysis and 
reasoning relying on multiple sources of evidence or multiple points of view; the DQP included intellectual skills in its 
model (Adelman et al., 2014; Jankowski et al., 2013). The Asia Society (2015) has prepared rubrics for educators to use 
in assessing students’ academic work in learning about global issues (including specifications of performance levels up to 
grade 12). The analytic skills elaborated in these rubrics include identifying evidence from different sources to address 
specific questions, integrating information from several sources into a coherent statement, and identifying counterargu¬ 
ments to a position. These rubrics form the basis of the Graduate Performance System (GPS). The guidelines set forth 
by the American Association of Community Colleges also describe intellectual skills such as identifying criteria for mak¬ 
ing judgments, evaluating and then defending a position on an issue, and judging the reliability of information sources 
(Gottlieb & Robinson, 2006). 

Analytic skills make a contribution to civic competency, particularly to the ability to understand and communicate 
to others about current civic and political conditions or events (as they are presented in publications or raised in 
discussions with others). Many of these analytic skills can be assessed by the presentation of written text, or politi¬ 
cal cartoons and graphic materials modeled on what appears in news media (in print or online), or in hypothetical 
scenarios, followed by appropriate questions. Additional examples of analytic skills can be found in Table 3. Many 
of these skills deal with seeing social and political problems with a realistic sense of their complexity. Among these 
skills is the individual’s ability to judge whether a statement is factual and based on evidence or a matter of opin¬ 
ion. The ability to track, evaluate, and compose arguments for and against a position is important. These skills also 
include perspective taking, or the ability to see positions on an issue from several points of view (including those of 
diverse groups). Finally, many disciplines are built upon skills that incorporate useful approaches to understanding 
and communicating about political and civic issues. The sample topics listed under analytic skills in the assess¬ 
ment framework found in Table 3 were distilled from the conceptual frameworks in Table 1 and the measures in 
Table 2. 

Participatory and Involvement Skills 

The participatory and involvement skills component of civic competency focuses on the ability to identify the most 
promising action in a group situation or in solving a social or civic problem. They include effective ways to listen to others’ 
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points of view and to mobilize others to take a public stand. Deliberation across difference as well as collaborative modes 
of decision making is emphasized by the National Task Force (2012). Soland, Hamilton, and Stecher (2013) in a Rand 
Corporation report on assessment, gave considerable attention to interpersonal skills, such as weighing other individuals’ 
perspectives and communicating effectively during collaboration. Likewise, the Asia Society’s GPS for Grade 12 includes 
rubrics for educators to judge students’ ability to collaborate across diverse groups, recognize alternative points of view, 
and tailor communications to specific audiences (Asia Society, 2015). The extent to which respondents have the knowl¬ 
edge of group process and the skills necessary to be an effective political and civic participant and leader in deliberative 
discussions across differences in culture and opinion should be assessed. These aspects of skills have been included in def¬ 
initions of civic competency relatively infrequently. However, the proliferation of service-learning experiences in higher 
education has been based, in part, on the belief that participating in activities involving members of the community can 
build students’ participatory and involvement skills. Possessing skills in participation and involving oneself in collective 
activities also contributes to the ability to be respectful and effective as a member of a campus group, a community group, 
or in a wider context. Understanding the real-life application of ethical principles forms an essential part of participatory 
and involvement skills. Individuals can also acquire skill in bringing the perspectives of disciplines that they have studied 
to bear on solving social problems. 

The rarity of measures of participatory and involvement skills as part of the assessment of civic competency can be 
traced in part to concerns about how to measure them. Self-ratings of such skills are limited in value (and subject to 
social desirability bias). Next-generation assessments present feasible options for more valid assessment of these skills. 
For example, many participatory and involvement skills could be assessed by the presentation of a scenario of group 
participation or of involvement with a community issue, followed by questions that ask the respondent to choose (and 
perhaps justify) the most effective strategies or actions (e.g., situational judgment items). More detailed examples of these 
skills and directions for assessment can be found in Table 3. 

Civic Engagement 

The second domain of the civic learning construct is civic engagement, which has three components: (a) motivations, 
attitudes, and efficacy, (b) democratic norms and values, and (c) participation and activities (see Table 3). Most of the 
constructs (Table 1) and assessments (Table 2) in this domain can be placed into these categories. Civic engagement can 
be described as active and informed practice or participation in democratic life (e.g., politically related behaviors, voter 
participation, volunteerism or service-learning, engagement in public action; Colby et al., 2007). 

Motivations, Attitudes, and Efficacy 

The first component of civic engagement—motivation, attitudes, and efficacy—refers to interest, involvement, or engage¬ 
ment in attending to political information along with the sense that one has the capacity to understand a political situation 
or undertake a successful civic or political action. The large majority of entries in Tables 1 and 2 mention this aspect of 
engagement. For example, the AAC&U VALUE rubric discusses the role of motivation and attitudes such as political 
efficacy as driving behaviors (political and nonpolitical) that promote the creation of change with the goal of improving 
an individual’s own civic life and the civic life of fellow community members (Rhodes, 2010). Other specific examples of 
motivations, attitudes, and efficacy can be found in Table 3. 

Democratic Norms and Values 

Democratic norms and values refers to the belief in basic principles of democracy (grounded historically and in the 
present) and to actions to foster a sense of respect in a diverse society. Important components are a sense of responsibility 
to engage in certain types of civic action and to avoid a sense of apathy. Although these beliefs are formulated differently 
across frameworks, Table 3 provides a number of examples. NASPA and ACPA (2004) identify both civic values and dis¬ 
positions as important components of civic engagement. Likewise, HERI includes civic values as part of its recommended 
student learning outcomes, using self-reported ratings of importance to measure the extent to which college students 
value political and social involvement as personal goals (e.g., helping others, promoting racial cohesiveness; Franke et al., 
2010). The Council for the Advancement of Standards in Higher Education (CAS) also includes social responsibility as 
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a dimension of humanitarianism and civic engagement, one of their six student learning and development outcomes 
(CAS, 2008). There are many other examples, especially associated with participation in service learning (e.g., IUPUI’s 
assessments). 

Participation and Activities 

Finally, relating to the third component of civic engagement, participation and activities refers to civic and political behav¬ 
ior and actions contextualized in a variety of settings. These range from face-to-face (on campus or in the community) 
to the national or global setting, and include online contexts (see Table 3 for specific examples). Existing frameworks 
and definitions have identified various civic activities such as volunteering or service learning, attentiveness to political 
news and respectful participation in political discussions, involvement in public action, participation in demonstra¬ 
tions, electoral involvement as a voter and/or as a campaign volunteer, actions demonstrating collective efficacy and 
facilitation of others’ civic engagement, community-based research and learning, involvement in organizations, online 
activism, and helping others in need (CAS, 2008; Franke et al., 2010; NASPA & ACPA, 2004; National Task Force, 2012; 
Rhodes, 2010). 

Assessment Design and Structure 

This section discusses item formats, task types, contexts, and accessibility considerations when designing a next- 
generation civic competency and engagement assessment. 

Item Formats 

Considering the multidimensional nature of civic learning, items in multiple formats should be employed for an adequate 
coverage of the two domains (see Table 4). A next-generation assessment of civic competency should consider a range of 
options. Multiple-choice items can be used to measure a wide range of factual and conceptual civic knowledge as well as 
the attainment of civic skills. Additionally, a variety of multiple choice and situational judgment items could be used to 
measure analytic and participatory and involvement skills. Situational judgment items can be enhanced through the use 
of technology. For instance, instead of reading a scenario, an examinee could watch a video of a scenario and then choose 
the appropriate response from the list of alternatives. 

Open-ended items allow for flexibility, allowing examinees to provide written or oral responses in their own words 
(e.g., Rhodes, 2010; Steinberg et al., 2011). Trained raters could score for quality of response such as accuracy/extent of 
problem definition, number of distinct actions or actors who could take action, understanding the role and limitations of 
institutions, ability to see constraints on solutions, and ability to tailor a solution to a context. These rubrics could also be 
the basis for computer-based scoring. This approach could be especially useful in assessing the extent to which students 
see social and political problems and their solutions in a realistic and complex way. See Bernstein (2010), Perrin (2006), 
and Torney-Purta (1992) for research examples. It is important to note, however, that open-ended items take longer for 
an examinee to complete and require the development of a scoring rubric. With restricted testing time and costs, it will 
be important to consider how many open-ended items would be feasible. 

Measuring an examinee’s level of civic engagement is different from assessing his or her level of civic competence and 
usually depends on self-report measures. The most common format for these measures is Likert-type items. These items 
can measure a variety of domains such as values (social responsibility), attitudes (toward specific issues such as diversity 
or participation), motivation (efficacy), perceived skill levels, perceived achievement, or competency and behaviors. With 
Likert-type items, it is important to consider respondents’ tendency to give responses that conform to perceived social 
norms. A recent study by Rios and Anguiano-Carrasco (2014) investigated the effect on scores on a low-stakes civic 
assessment of respondents’ not providing truthful answers (i.e., which they referred to as faking). The distortion was 
about 0.27 to 0.50 SDs; this is less than the distortion reported earlier for personality or employment tests but is still of 
concern. These results point to the need to consider respondents’ tendency to provide a socially desirable answer on some 
Likert-type items. The issue may be of more concern when the stakes for the assessment results are high. One way to 
address this issue would be to require respondents to provide written justification in the form of examples illustrating or 
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Table 5 Examples of Task Types for Assessing Civic Competency 


Task type 

Description 

Analyze a document/argument 

Examinee reviews an existing document, argument, or graphic before 
answering a question 

Conflict resolution 3 

Examinee provides information about alternative ways to solve a conflict 
in various contexts 

Draw conclusions 

Examinee draws inferences from information provided or extrapolates 
additional likely consequences 

Deliberation 3 

Examinee provides information about how to intervene/deliberate in a 
political debate or discussion in a way that furthers productive 
discussion 

Fact checker/recognize bias 

Examinee reviews and analyzes facts and opinions, recognizing 
misleading information and facts from opinions (or whether a 
statement is biased against certain groups) 

Generating critical questions 3 

Examinee develops or evaluates queries to elicit information to evaluate 
an argument or claim 

Identify compelling evidence 

Examinee recognizes evidence statements with the conclusions they 
support or undermine 

Justification (based on response to a self-report item) 3 

Examinee provides rationale for a previous response to a self-report item 
(e.g., Likert-type or short answer) 

Perspective taking 3 

Examinee role plays, takes perspectives, or chooses which response is the 
best choice for particular “participants” or stakeholders with 
contrasting resources and/or goals 

Using the past to predict/inform the present 

Examinee uses historical/previous information to provide justification for 
a response to a stimulus 

Knowledge application 

Examinee analyzes knowledge presented in a table or graph (or other 
source) to answer a question or solve a problem 


“These tasks could also be used in measuring civic engagement. 


validating their responses to some of the Likert-type questions. Even if no rubrics were developed for scoring this open- 
ended material, respondents should be less likely to inaccurately report socially desirable activities if they knew they might 
be asked to provide specific examples or elaborations. Additionally, alternative item formats, such as forced-choice items, 
could potentially mitigate respondents’ tendencies to respond in a way that makes them appear more civically engaged 
than they actually are. 

Task Types 

A number of task types can be used to assess civic competency (see Table 5). For instance, tasks could include recog¬ 
nizing the most compelling evidence regarding a civic problem solution, recognizing inconsistency and bias in political 
media reports, generating critical questions to ask based on a scenario, or analyzing an argument in a mock media report 
(based on Table 6 in Liu, Frankel, & Roohr, 2014; Torney-Purta et al., 2001). Tasks can be constructed using adaptations 
of published media reports, graphs, or cartoons. Another task could include the ability to take the perspectives of dif¬ 
ferent individuals in a situation or a problem-solving scenario item about group participatory skills where students were 
presented with more and less democratic approaches to arriving at a decision in a group, alternative ways to arrive at 
consensus, or alternative ways to productively engage in disagreement. 

In addition to measuring civic competency, certain tasks can also be used to measure civic engagement (see 
Table 5). When using tasks such as these to measure civic engagement, it is critical to think about the combina¬ 
tion of the task and the item format. An item format and task combination that could be used to measure civic 
engagement is a self-report item (e.g., Likert-type or short answer) with an open-ended justification. The open-ended 
format could give an examinee the opportunity to justify a previous response to a self-report item. For instance, if 
an examinee reported participating in five civic-related activities, the justification would be listing several of those 
activities. 


38 


ETS Research Report No. RR-15-34. © 2015 Educational Testing Service 





J. Torney-Purta etal. 


Assessing Civic Competency and Engagement 


The United Kingdom’s GCSE tests in citizenship studies have a number of tasks that could be considered as prototypes 
or extensions appropriate for a next-generation civic competency and engagement assessment. Some of these would be 
short essays scored with rubrics, but others could correspond to the “monitored exercise” in which students engage in 
projects that are supervised by a teacher. In the United States, a number of disciplines (e.g., psychology, political science, 
sociology, and economics) are requiring the documentation of capstone experiences by college seniors. Some scholars 
are suggesting using this documentation for both examining individual learning and at institutional levels of evaluation 
(Hauhart & Grahe, 2012; Reason, 2011; Sum & Light, 2010). This could be an extension attractive to some institutions. 

Scoring Considerations 

In addition to suggesting item formats and task types, it is also important to identify how items could be scored. For an 
assessment measuring civic competency and engagement, an important distinction exists between providing a score for a 
civic competency versus providing a score for civic engagement. A large proportion of the item types used to measure civic 
competency could be scored for a correct answer. When administered on a computer, scores could be derived automati¬ 
cally. For open-ended questions, there is potential to score them using automated scoring tools. For example, automated 
scoring has been used to score science content (Liu, Brew, et al., 2014), mathematics content (Sandene, Horkay, Bennett, 
Braswell, & Oranje, 2005), writing quality (Burstein & Marcu, 2003), and speech (Higgins, Zechner, Xi, & Williamson, 
2011). However, to our knowledge, such applications have not been extended to scoring an assessment of civic compe¬ 
tency. More empirical evidence is required to determine the accuracy of using automated engines to score items with civic 
content. 

Items used to measure civic engagement are typically self-report and would in almost all cases not be scored as right 
or wrong. As a result, a “score” for civic engagement would be someone’s level of behaviors or attitudes associated with 
engagement (which could be compared to averages developed from groups of students). Future research should consider 
evaluating the association between the scores in the two domains. 

Contexts 

When developing a next-generation civic competency and engagement assessment, it is important to consider the con¬ 
text or situation in which the tasks are embedded. Contexts can be divided into two main areas: level and setting. Level 
refers to whether the context of those items is at the campus, local community, national, or global level. As previously 
discussed, specifying the setting of the assessment is a challenge when developing a next-generation civic competency 
and engagement assessment suitable for all types of higher education institutions. It is recommended that the national 
and global contexts include contemporary or historical assessment tasks and that the local community context focus on 
contemporary issues. 

The next important contextual area is the setting, which includes the workplace, institution (i.e., a campus organi¬ 
zation), community/neighborhood (e.g., volunteering or service learning organization), and political organizations or 
institutions. Diversity within these various settings is important to consider (and may differ between residential and com¬ 
muter institutions). Online or virtual settings are also critical to consider, given globalization (including the growing 
number of international corporations and the expansion of communication media). For example, major technological 
advances such as smartphones and tablets have substantially increased information exchange. Individuals’ mobility has 
also increased. These changes have propelled major initiatives that involve international, intercultural, and multinational 
awareness, competence, and cooperation, as well as conflict (Coelen, 2013). It is also the case that online civic-related 
communication can be of different types; for example, according to Kahne et al. (2013), communication may be driven 
by one’s personal political ideology, by interest in a particular social or political issue (either expressing an opinion or 
seeking information), or by a desire to initiate or maintain a relationship with someone who reads the communication. 
The Crucible report and actions of the National Task Force (2012) also recognized these trends. 

Delivery Modes and Accessibility 

According to the Standards for Educational and Psychological Testing, “standardized tests should be designed to facilitate 
accessibility and minimize construct-irrelevant barriers for all test takers in the target population, as far as practicable” 
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(AERA et al., 2014, p. 57). Given the changing demographics in higher education, a next-generation assessment of civic 
competency and engagement should aim to provide access for all students, including those with disabilities and English 
learners (ELs), through a universal design. Universal design refers to the “design of products and environments to be 
useable by all people, to the greatest extent possible, without the need for adaptation or specialized design” (Measured 
Progress/ETS Collaborative, 2012, p. 4). This means all students in the intended testing population, “regardless of char¬ 
acteristics such as gender, age, language background, culture, socioeconomic status, or disability” (AERA et al., 2014, 
p. 57). In the case of a next-generation civic competency and engagement assessment, universal design means design¬ 
ing tasks for a broad range of students and providing item adaptations for students with special access needs. Ideally, if 
universal design is appropriately applied, a minimal number of item adaptations are needed (Measured Progress/ETS 
Collaborative, 2012). For instance, although political cartoons could serve as stimulus material for test questions, an 
assessment developer would need to make sure that the cartoon would be accompanied by a detailed description to be 
used with visually impaired students. It may be possible to develop separate test forms that are accessible. Addition¬ 
ally, for ELs, it is important to reduce the number of complex English phrases that could result in construct-irrelevant 
variance. 

Even when universal design is applied to assessment development, there are still situations where the instru¬ 
ment might not be appropriate for all students, and as a result, test adaptations would need to be made (AERA 
et al., 2014). Although paper-and-pencil tests are one method of delivery, computer-based assessments allow 
for more flexibility in item-level adaptations. For instance, a screen-reader could be put in place for visually 
impaired students. Additionally, a computer-based assessment could allow for on-demand font magnification. 
While technology could help to improve accessibility for all students taking an assessment, we must also make 
sure that technological literacy does not become a source of construct-irrelevant variance, especially for students 
who may not have extensive experience with technology. This means providing tutorials about how to navigate 
through the computerized test administration to make sure the examinees are familiar with the layout and item 
formats. 


Potential Advantages of the Proposed Framework and Assessment Considerations 

Several distinguishing features of this proposed framework and the associated assessment considerations provide 
advantages over previous approaches. First, the proposed framework distinguishes between two important civic learning 
domains: civic competency and civic engagement. These two domains are defined based on a review and synthesis 
of existing frameworks, definitions, and assessments of civic learning, and both domains could be incorporated in an 
assessment. A framework that captures both civic competency and civic engagement as part of civic learning is rare in 
higher education. Second, this framework would be useful for a range of institutions from community colleges to 4-year 
institutions of various types (e.g., public or private institutions). It would also be of interest to several disciplines as well 
as to groups that foster interdisciplinary collaboration. Third, this framework has been designed taking into account 
psychometric considerations and suggesting next-generation assessment approaches. Assessments could be carefully 
designed to assess the multidimensional constructs of civic competency and engagement, employing alternative item 
formats such as forced-choice or situational judgment items. In addition to item formats, we also discussed a classification 
of task types that could be used to guide assessment development. The specification of these assessment considerations 
helps to clarify how the proposed framework can be translated into a next-generation assessment. Lastly, this framework 
also recognizes the importance of universal design and the use of technology to make an assessment accessible for 
students with disabilities or ELs. 


Conclusion 

It is an excellent time to explore the development of an assessment of civic competency and engagement for college and 
university students. A variety of higher education associations and institutions are taking steps in this direction (e.g., 
developing frameworks, institutionalizing conceptualizations, and thinking about the need for assessments and ways 
to recognize students’ achievements as well as shortfalls in this area). There are approximately 30 entries in Tables 1 
and 2 that discuss projects relevant to higher education involving civic-related constructs. New test development 
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technologies (e.g., online and with video links) and methodologies make this effort much more feasible than was once 
the case. 

A variety of stakeholders extending beyond universities and their accrediting agencies, such as leaders in the workforce 
community, have an interest in this topic and are potential sources of support for such an effort. Many institutions want 
to demonstrate that they are preparing students for the workplace and citizenship. Students themselves want to have 
validation and recognition for their civic-related activities taking place in settings such as campuses, local communities, 
the workplace, national organizations, political structures, international or global contexts, and online. This area will be of 
interest for all college majors, including students in STEM majors. Many employers are likely to have an interest in ways to 
assess the civic-related capacities of their future workers. This includes abilities to take the perspective of other people, to 
understand diverse groups, and to formulate workable solutions to complex social issues. Administrators of colleges and 
universities want to give information about students’ achievement to faculty members in a way likely to improve programs. 
A project in this area may have special utility because civic learning does not neatly fit into a disciplinary category. Finally, 
a next-generation assessment of civic competency and engagement could be more informative than counting how many 
students are participating in service learning or voter registration drives. And to the extent that this overall effort simulates 
further engagement in the local community, it is likely to be welcomed as a way to integrate college students during and 
after their postsecondary studies. 
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